Prepare listing JSON and stage isolated database capacity - #1756

Cyclic listing bursts hold database connections while application decoding competes with ingress and PostgreSQL for CPU. This change closes listing rows before typed decoding, prepares versioned listing JSON during writes, and adds checkout timing, opt-in loopback profiling and configurable per-replica pools. Derived JSON preserves metadata, invalidates on old-writer updates, falls back for unknown codecs and has a bounded backfill tool.

Production acceptance is still pending. The new placement is running in staging; production has not changed.

Staging capacity and placement

The staging-only profile uses one fixed, tainted e2-standard-4 database node and two–three e2-standard-4 workers: three total idle nodes, four under load. PostgreSQL requests its full 3-CPU limit. App and ingress require worker nodes and retain per-revision hostname spread. Apps scale 3–4 with 800m requests and pools 30/5; ingress scales 3–6 with 800m CPU / 512Mi memory requests. Both CPU targets are 70%, with conservative scale-down. The prepared-listing staging image is digest-pinned. Production enablement of this experimental profile is rejected.

PostgreSQL, three apps and three ingress replicas rolled out successfully. All 48 saved response fixtures matched; the corpus remained 52,366 names / 187,108 versions with zero invalid indexes. One of 84 external health probes failed during rollout, so this was not an interruption-free transition.

At 13:12:29 UTC on October 29, staging naturally consolidated to one DB node and two workers, with all apps and ingress Ready: 2 apps + 1 ingress on one worker, 1 app + 2 ingress on the other. All 20 health probes during idle observation succeeded. No forced eviction or resize was used. This proves idle placement/consolidation, not loaded burst recovery. System agents still share the DB node; PostgreSQL remains a singleton. Node-upgrade surge can temporarily exceed four nodes. Public networking was not changed.

Staging deployment transition

The database pool and app HPA already exist from the staging experiment. Before the normal staging apply, GitHub Actions will use its existing staging passphrase secret to import these exact resources into Pulumi. The adoption script skips completed imports, checks stack/provider/resource identities and rejects conflicting ownership. The DB pool is protected from deletion. Staging apply refreshes state before planning.

Local adoption was blocked by the unavailable stack passphrase. The script's plan was checked against the exported staging state, but the CI import/apply has not yet executed. The pinned staging image deliberately preserves the tested candidate until acceptance; remove that override when returning to normal staging builds.

Existing performance evidence and limits

The same ordered 274,116-read production trace was compared across baseline, app only, and the earlier shared-node infrastructure candidate. App-only sampled CPU fell approximately 31% (2.13 to 1.47 cores). Generator drops fell from 2.10% to 0.84%; attempted page p95 fell from 3,322 to 1,597 ms. The older infrastructure arm dropped 0.70% and had seven transport failures. All arms failed the short delivery gate.

Drops were confined to the initial 84 seconds of an abrupt replay into empty ingress caches. Twelve final-minute cache/body-size/key-diversity/query-mix checks passed, but this does not qualify the startup cache state or full production cycle. Detail 404s from the older staging corpus and single-source rate-limit 429s must be resolved in the next comparison.

Earlier recovery completed 180,000/180,000 reads, but that was a different two-replica/shared-node candidate. Its result is not credited to this placement. The ten-minute peak and publish workload have not yet been tested with the new layout.

Validation and promotion gate

• Deployment-module Go tests and changed deployment-code lint passed.
• Five adoption tests passed, covering retries and rejection of wrong stack, identity, provider or duplicate ownership; workflows/configuration parsed as YAML.
• Existing application, benchmark, race and workload-fidelity validation is retained in the linked reports.
• Before production promotion: replay preceding observed traffic to establish cache state, then the selected peak with publish traffic and recovery. Count all offered work, including generator drops; verify per-replica sharing, CPU pressure, connection waits, errors/latency and natural return to three nodes. Keep cold-cache startup as a separate resilience test.

See isolated placement and evidence, production trace comparison, workload audit and capacity candidate.